Image processing system that computerizes document, control method thereof, and storage medium

ABSTRACT

To make it easy for a user to grasp with which text block within a preview pane, a setting item within a property pane is associated. By using a color allocated to a setting item displayed in the property pane, at least a part of a display field corresponding to the setting item is colored. Then, in a case where a user selects one of text blocks in a scanned image preview-displayed in the preview pane in the state where one of setting item is selected in the property pane, the text block is colored by using a color allocated to the setting item in the selected state.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation Application of U.S. patentapplication Ser. No. 17/153,075, filed on Jan. 20, 2021, and whichclaims the benefit of Japanese Patent Application No. 2020-007518, filedon Jan. 21, 2020, and Japanese Patent Application No. 2020-031689, filedon Feb. 27, 2020, the entire disclosures of which are all herebyincorporated by reference herein.

BACKGROUND Field

The present disclosure relates to a graphical user interface (GUI) atthe time of computerizing a document and making use of the computerizeddocument.

Description of the Related Art

Conventionally, it has been performed to extract a character string byperforming optical character recognition processing (OCR processing) fora scanned image obtained by scanning a business form, such as an ordersheet, and make use of the character string as a file name or the like.Japanese Patent Laid-Open No. 2019-115011 has disclosed a technique toautomatically generate a file name by extracting a character stringincluding a specific key from character strings included within adocument image and present the file name to a user along with a previewdisplay of the document image. With this technique, it is possible for auser to easily set a file name while checking the document image. Then,it is also performed to save and manage a character string extractedfrom a scanned image as metadata and make use of the metadata as a filename or the like.

As the configuration of a UI screen of an application at the time ofsetting a file name or metadata by using a recognized character stringobtained by OCR processing, a two-pane configuration including a previewpane and a property pane is well known. Here, the preview pane is anarea for preview-displaying a computerization-target scanned image andthe property pane is an area for displaying contents of a property, suchas a file name and metadata, in a list. In a case where a document, suchas an order sheet or a bill, is computerized, a plurality of characterareas (text blocks) is included in the scanned image and it is possiblefor a user to visually recognize each text block by the preview pane.Further, by the property pane, it is possible for a user to check thecontents of a variety of setting items, such as “Customer Name”, “Date”,and “Address”. For example, it is assumed that a user selects a textblock of “XYZ Corporation” within the preview pane for a setting item of“Customer Name” within the property pane on the above-described two-paneconfiguration UI screen. As a result of that, the recognized characterstring “XYZ Corporation” within the scanned image is saved inassociation with the setting item “Customer Name”. In this case, on acondition that a plurality of text blocks exists within the preview paneand a plurality of setting items exists also within the property pane,it is difficult for a user to immediately grasp which block isassociated with which setting item.

The present disclosure has been made in view of the above-describedproblem and an object is to make it possible for a user to easily graspthe correspondence relationship between a text block and a setting itemon a UI screen having a preview pane and a property pane.

SUMMARY

The image processing system that computerizes a document according tothe present disclosure includes: a memory that stores a program; and aprocessor that executes the program to perform: displaying a userinterface screen for setting a property to a scanned image obtained byscanning the document; and controlling a display on the user interfacescreen, and on the user interface screen, at least a preview pane thatis an area preview-displaying the scanned image and a property pane thatis an area to which a setting value of each setting item of the propertyis input exist, and wherein at least a part of a field of a setting itemdisplayed in the property pane and a text block corresponding to thesetting item in the scanned image preview-displayed in the preview paneare colored with a same color or similar colors.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an entire configuration of an imageprocessing system;

FIG. 2 is a block diagram showing a hardware configuration of aninformation processing apparatus;

FIG. 3 is a diagram showing a software configuration of the imageprocessing system;

FIG. 4A is a diagram explaining a state transition of a UI screen;

FIG. 4B is a diagram explaining an orthogonal coordinate system;

FIG. 5 is a diagram explaining a state transition of a UI screen;

FIG. 6 is a flowchart showing a flow of UI screen control processingaccording to a first embodiment;

FIG. 7 is a diagram explaining a problem of a modification example 2;

FIG. 8 is a diagram explaining effects of the modification example 2;

FIG. 9 is a flowchart showing details of coloring control processingaccording to the modification example 2;

FIG. 10A is a diagram explaining similarity determination of a color andFIG. 10B is a diagram explaining a change in an item color;

FIG. 11 is a flowchart showing details of coloring control processingaccording to a modification example 3;

FIG. 12 is a diagram explaining effects of the modification example 3;

FIG. 13 is a flowchart showing a flow of UI screen control processingaccording to a second embodiment;

FIG. 14 is a diagram explaining a state transition of a UI screen; and

FIG. 15 is a diagram explaining a state transition of a UI screen.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, with reference to the attached drawings, the presentdisclosure is explained in detail in accordance with preferredembodiments. Configurations shown in the following embodiments aremerely exemplary and the present disclosure is not limited to theconfigurations shown schematically.

First Embodiment <System Configuration>

FIG. 1 is a diagram showing the entire configuration of an imageprocessing system that computerizes a document according to the presentembodiment. The image processing system includes a scanned documentprocessing server 111, a client terminal 121, and a work server 131. Theclient terminal 121 is connected with the scanned document processingserver 111 and the work server 131 so as to be capable of communicationvia a network 101, such as an intern& and an intranet. The clientterminal 121 may be an MFP (Multifunction Peripheral) and the likecomprising the scanner function, in addition to a personal computer, alaptop computer, a tablet computer, a smartphone and the like. Theconfiguration of the image processing system shown in FIG. 1 is anexample, but the configuration is not limited to this. For example, itmay also be possible for the client terminal 121 to have the function ofthe scanned document processing server 111. In the followingexplanation, a scanned image obtained by scanning a processing-targetdocument, such as an estimate form, is called “scanned document”.

<Hardware Configuration>

FIG. 2 is a block diagram showing the hardware configuration of thescanned document processing server 111, the client terminal 121, and thework server 131, each as an information processing apparatus. A userinterface (UI) 201 is implemented by a display, a keyboard, a mouse, abutton, a touch panel and the like and inputs and outputs informationand signals. It is also possible for a computer not comprising thehardware such as this to connect and operate from another computer by aremote desktop, a remote shell, and the like. A network interface 202connects to the network 101 and performs communication with anotherinformation processing apparatus and a network device, not shownschematically. The communication method may be wired or wireless. A CPU203 is a computing device in charge of the control of the entireapparatus and executes programs read from a ROM 204, a RAM 205, asecondary storage device 206 and the like. The ROM 204 storesincorporated programs and various kinds of data. The RAM 205 temporarilystores various kinds of data and also functions as a work memory of theCPU 203. The secondary storage device 206 is a large-capacity storagedevice represented by an HDD and a flash memory. Each of the unitsdescribed above is connected to one another via an input/outputinterface 207.

<Software Configuration>

FIG. 3 is a diagram showing the software configuration of the imageprocessing system according to the present embodiment. Each piece ofsoftware is installed in the scanned document processing server 111, theclient terminal 121, and the work server 131, respectively, describedabove and executed by the CPU 203 of each apparatus. In the following,each piece of software is explained.

<<Software of Scanned Document Processing Server>>

A scanned document processing application 311 is a Web applicationinstalled in the scanned document processing server 111. However, thescanned document processing application 311 may be in an applicationmounting form other than a Web application. An API 312 is an API(Application Programming Interface) provided by the scanned documentprocessing application 311. A Web UI 313 is a Web user interfaceprovided by the scanned document processing application 311.

A data store 321 is a storage area saving and storing data that is usedby the scanned document processing application 311 or a backendapplication 331, to be described later. In the data store 321, a scanneddocument storage unit 322, a scanned document job queue 323, a propertymanagement unit 324, and a scanned image processing results storage unit325 exist. The scanned document storage unit 322 files the data of ascanned image itself and saves as an image file, such as JPEG, or adocument file, such as PDF (Portable Document Format). The scanneddocument job queue 323 stores a queue managing jobs to be subjected toproperty input processing, to be described later. The propertymanagement unit 324 manages a list of setting items and a format of aname or a value (character string, figure and the like) for each settingitem, relating to properties, such as a file name, a folder path, andmetadata, which need to be attached for each scanned document, itemcolor information, to be described later, and the like. The scanneddocument processing results storage unit 325 stores OCR processingresults and business form determination results. Further, the scanneddocument processing results storage unit 325 stores character areainformation (text block information) associated with a setting item, aset value, and the like for each scanned document.

The backend application 331 is an application for performing backgroundprocessing. The backend application 331 is an application in charge ofvarious kinds of processing that may be performed sequentially on thebackground, specifically, OCR processing, business form processing, andcommunication processing. An OCR processing unit 332 acquires aprocessing-target scanned document from the scanned document storageunit 322 and performs OCR processing for the acquired scanned document.In the OCR processing, information specifying each text block(specifically, information on the start point coordinates, width, heightof a rectangular area) and information on a recognized character stringof each text block are extracted. A business form processing unit 333determines the type of the scan-target document (business form) by usingthe information on the processing-target scanned document, thearrangement pattern of text blocks and the recognized character strings,which are obtained by the OCR processing. This determination processingmay be any method, such as pattern recognition and machine learning. Acommunication processing unit 334 performs processing to transmit ascanned document and analysis processing results thereof to the workserver 131, which is an external apparatus. In a case where it is notnecessary to transmit a scanned document and analysis processing resultsthereof, the communication processing unit 334 may be omitted.

<<Software of Client Terminal>>

A client application 351 performs the Web application of theabove-described scanned document processing application 311. Forexample, the client application 351 performs the Web application bydisplaying the Web UI 313 by a Web browser and performing transmissionand reception of necessary data with the API 312.

<<Software of Work Server>>

A work application 361 is an application for assisting various kinds ofwork in a company, such as file management, document management, orderreception, and accounting. A work data storage 362 saves various kindsof data used by the work application 361. The work application 361receives results of processing by the scanned document processing server111 and performs necessary processing for the results and stores theresults.

<UI Screen of Client Application>

Following the above, a user interface screen (hereinafter, described as“UI screen”) that is displayed on the user interface 201 of the clientterminal 121 is explained. FIG. 4A shows an example of a UI screen thatis displayed on the user interface 201 by the client application 351according to the present embodiment.

The UI screen shown in FIG. 4A has a two-pane configuration of a Previewpane 401 and a Property pane 411. Here, within the Preview pane 401, apage image 402 of a scanned document of “PURCHASE ORDER” ispreview-displayed. In the Preview pane 401, it is possible to displayany position in the page image 402 by scrolling or zooming the pageimage 402. The Property pane 411 is a pane for displaying and editing aproperty to be attached to the processing-target scanned image. Itemsthat are set as properties include master items, such as “File Name”,“Folder Path”, and “Metadata”, and slave items, such as “Company Name”,“Date”, and “Address”, which are elements configuring the master items.Explanation below is given by taking a case as an example where“Metadata” is set and registered as the master item.

The UI screen in FIG. 4A is an example in a case where the business formtype of the processing-target scanned document is determined to be“PURCHASE ORDER” in the business form determination processing. Then,the state is such that three slave items “Customer Name”, “Address”, and“Billing Name” are displayed as the contents of the master item“Metadata” to be attached to the scanned document. In the followingexplanation, the slave item of the metadata is called “metadata item”.

FIG. 4B is a diagram explaining an orthogonal coordinate systemindicating a relationship between the page image 402 and each textblock. In FIG. 4B, three text blocks 421, 422, and 423 exist. In thepage image 402, there are more text blocks, but they are omitted here.The text block detected by OCR processing is identified respectively bythe start point coordinates, width, and height as shown by a hatchedrectangle. For example, the text block 423 corresponding to thecharacter string of “XYZ Corporation” is represented as “Start pointcoordinates (1200, 700)”, “Width: 720” and “Height: 120” in units ofpixels.

FIG. 5 is a diagram explaining the way a text block within the pageimage 402 is associated with the metadata item on the UI screen in FIG.4A. Here, the state is such that a text block 501 corresponding to thecharacter string of “XYZ Corporation” within the page image 402 and themetadata item “Customer Name” (item of 510 a) are associated with eachother. Then, in order to enable a user to easily identify, on the textblock 501, a rectangle colored in a color (in the example in FIG. 5,“red color” allocated to the metadata item “Customer Name”) allocated toeach metadata item based on item color information is displayed in atransparent and superimposing manner. The coloring method is not limitedto the transparent and superimposed display and may be a superimposeddisplay or the like by a colored frame line. Horizontally elongatedrectangles 510 a to 510 c within the Property pane 411 are displayfields in which setting values and the like of the metadata items, whichare slave items, are displayed and edited for the metadata, which is themaster item. Further, vertically elongated rectangles 511 a to 511 c areitem color areas that are each provided at a portion within the displayfield and which each display a color (item color) allocated to eachmetadata item so as to enable a user to identify the color. Then,horizontally elongated rectangles 512 a to 512 c are each an area(cut-out image area) in which an image (image cut out from the pageimage 402) corresponding to the text block selected by a user isdisplayed. In the stage in which association with any text block is notperformed yet, the state is such that a “+” button is displayed as inthe cut-out image areas 512 b and 512 c and the item colors of the itemcolor areas 511 b and 511 c are not displayed. Horizontally elongatedrectangles 513 a to 513 c located under each of the cut-out image areas512 a to 512 c are each an editing area in which the recognizedcharacter string included in the text block relating to user selectionis displayed and it is possible for a user to edit the displayedcontents with a soft keyboard or the like, not shown schematically.

<UI Screen Control Processing>

Next, control processing to associate a text block with each metadataitem on the UI screen described above, which is performed by the clientapplication 351 of the client terminal 121, is explained by using aflowchart in FIG. 6. Prior to the start of execution of the flow shownin FIG. 6, first, a queue of jobs to be processed is provided to theclient application 351 by the scanned document processing application311 of the scanned document processing server 111. In a case where auser selects a job of a scanned document for which the user desires tostart processing by using the client application 351, the scanneddocument processing application 311 acquires necessary informationcorresponding to the selected scanned document from the data store 321.Specifically, the scanned document processing application 311 acquiresthe corresponding image file from the scanned document storage unit 322and acquires the corresponding OCR processing results and business formtype determination results from the scanned document processing resultsstorage unit 325. Further, the scanned document processing application311 acquires the metadata item list for each business form type and theitem color information for identifying a predetermined color allocatedin advance for each metadata item from the property management unit 324.These pieces of acquired data are provided to the client application 351from the scanned document processing application 311 and the flow inFIG. 6 is started. In the following explanation, symbol “S” represents astep.

At S601, the data of the page image and the OCR processing results(recognized character string) is loaded, which are to be displayedwithin the Preview pane 401. At S602 that follows, the metadata itemlist corresponding to the business form type and the item colorinformation are loaded. At next S603, the contents of the data loaded atS601 and 5602 are reflected on the UI screen. Specifically, the pageimage is displayed within the Preview pane 401 and each metadata itemincluded in the metadata item list is displayed within the Property pane411. By the processing at S601 to S603 performed so far, the state ofthe UI screen shown in FIG. 4A is brought about. At this point in time,coloring to the item color areas 511 a to 511 c of each metadata itemwithin the Property pane 411 is not performed yet and the item colorareas 511 a to 511 c are in the default (for example, gray) state. Thereason is to present that metadata is not set yet to a user in aneasy-to-see manner and prompt the user to set metadata by representingvisually that coloring is not determined yet, together with the “+”button display in the cut-out image area 512 described previously.

Then, at S604, an operation event of the selection operation of the “+”button of the cut-out image areas 512 a to 512 c in one of the metadataitems within the Property pane 411 is detected. This selection operationis an operation of a user touching or the like the “+” button on thetouch panel with a finger or a touch pen, or a click operation with amouse. In this manner, one metadata item relating to the user selectionis determined among one or a plurality of metadata items included in themetadata item list. In the following, the metadata item selected by auser is called “selected metadata item”. At S605 that follows, the itemcolor area of the selected metadata item determined at S604 is coloredin accordance with the item color information loaded at S602. Here, itis assumed that in the item color information, a specific color isassociated with the order of each slave item existing within the masteritem. For example, four colors of blue, green, yellow, and red aredefined in advance as usable colors and the item order and the color areassociated with each other in advance, such as that the first settingitem is associated with “blue”, the second setting item with “green”,the third setting item with “yellow”, and the fourth setting item with“red”. In this case, the three slave items of “Customer Name”,“Address”, and “Billing Name” of the master item “Metadata” are in thisorder, and therefore, “blue”, “green”, and “yellow” are allocated,respectively, as the item colors. By associating a specific color withthe order of a plurality of setting items as described above, it ispossible to make common the item color information in a variety of kindsof setting item. It may also be possible to allocate in advance aspecific color to each individual specific setting item in the itemcolor information. Further, by highlighting the display field of theselected metadata item for which the selection operation has beenperformed by a user, for example, such as changing the color of the areaother than the item color area within the display field, it is madeeasier for a user to identify the metadata item currently beingselected. As a method of making it possible to identify the metadataitem currently being selected is not limited to the method of changingthe color, and for example, it may also be possible to highlight thedisplay field of the metadata item by surrounding the display field witha thick-line frame.

Then, at S606, an operation event is detected, such as a touch operationto one of text blocks existing in the page image 402 within the Previewpane 401. In this manner, one arbitrary text block is selected from oneor a plurality of text blocks included in the page image. In thefollowing, the text block selected by a user is called “selected textblock”. At S607 that follows, the selected text block determined at S606is colored in the same color as that in which coloring has beenperformed to the item color area within the selected metadata itemcolored at S605. Then, at S608, the cut-out image corresponding to theselected text block colored at S607 is displayed in the cut-out imagearea within the selected metadata item. Further, the recognizedcharacter string included in the selected text block colored at S607 isdisplayed within the editing area of the selected metadata itemdetermined at S604. By the processing at S604 to S608 performed so far,the state of the UI screen as shown in FIG. 5 is brought about. That is,the metadata item selected by a user and the text block are associatedwith each other and which text block is used to input the characterstring used in the selected metadata item is determined.

In a case where the above processing is completed for all the metadataitems within the metadata item list and a “Register” button 514 istouched by an operation of a user, at S609, a request to registermetadata is transmitted to the scanned document processing server 111.In this metadata registration request, information associating editedvalues in all the metadata items and the text blocks with each other isincluded. In the scanned document processing server 111, the scanneddocument processing application 311 saves the above-describedinformation included in the metadata registration request received fromthe client terminal 121 in the scanned document processing resultsstorage unit 325. At this time, in a case where the registration ofmetadata is performed in the external work server 131, the metadataregistration request is transferred to the work server 131 via thecommunication processing unit 334. After all the necessary processing iscompleted, the scanned document processing application 311 dequeues thequeue from the scanned document job queue 323. In this manner, thisprocessing is completed.

The above is the contents of the UI screen control processing in theclient application 351 according to the present embodiment. In thepresent embodiment, the item color area indicating the color allocatedfor each setting item is provided for each display field of each settingitem, but it is not mandatory to provide each individual item colorarea. That is, it may also be possible for the area (area except for thecut-out image area and the editing area in each display field) for thehighlight indicating that the setting item is selected by a user todisplay an item color. In a case where the configuration is made so thatthe item color is displayed in the area except for the cut-out imagearea and the editing area within each display field, as the highlightindicating that a use has selected the setting item, it is sufficient toproduce the highlight by, for example, attaching a thick-line frame thatsurrounds the selected display field, changing the density of the color,and so on.

MODIFICATION EXAMPLE 1

In the flow in FIG. 6 described above, at the timing (S605) of thehighlight accompanying the selection of the metadata item of interest bya user (detection of touch operation to the “+” button within thecut-out image area), the coloring to the item color area is performed.However, the timing of coloring to the item color area is not limited tothis. For example, it may also be possible to perform coloring to theitem color area within the selected metadata item at the same timing asthat of coloring to the selected text block at S607.

MODIFICATION EXAMPLE 2

In the flow in FIG. 6 described above, by coloring the selected textblock within the Preview pane in the same color as that of the itemcolor area of the selected metadata item, it is made possible for a userto grasp the correspondence relationship between the metadata item andthe text block. At this time, there is a possibility that the color usedfor coloring and the character color or the background color within theselected text block are similar to each other. In this case, the visualrecognizability of the color used for coloring to the selected textblock is degraded. Consequently, an aspect is explained as amodification example 2 in which the color used for coloring is changedas needed.

FIG. 7 is a diagram explaining a problem of the present modificationexample, which corresponds to FIG. 5 described previously. In a pageimage 402′ in FIG. 7, three character strings “XYZ Corporation”, “1Pumpkin Road, New York, N.Y.”, and “(123) 456-7890” whose charactercolor is black are surrounded by ruled lines and the background color inthe light blue color is attached to the inside thereof. Here, the stateis such that a text block 701 corresponding to the character string of“XYZ Corporation” within the page image 402′ and the metadata item“Customer Name” are associated with each other. Then, among the threemetadata items, “Customer Name” is the first setting item, andtherefore, the blue color is displayed in the item color area 511 awithin the display field 510 a thereof. As a result of that, the lightblue color, which is the background color within the ruled linessurrounding the character string of “XYZ Corporation”, and the bluecolor used for coloring to the text block 701 are similar to each other(similar colors). In this situation, it is difficult for a user to graspthat the text block 701 and the metadata item “Customer Name” areassociated with each other.

FIG. 8 is a diagram explaining effects in a case where the presentmodification example is applied. As in FIG. 7, the state is such thatthe text block 701 corresponding to the character string of “XYZCorporation” within the page image 402′ and the metadata item “CustomerName” are associated with each other. However, in FIG. 8, in the itemcolor area 511 a of the metadata item “Customer Name”, the red color isdisplayed in place of the original blue color. As a result of that,between the light blue color, which is the background color within theruled lines surrounding the character string of “XYZ Corporation”, andthe red color in which the text block 701 is colored, a contrast incolor occurs, and therefore, it is made possible for a user to easilygrasp the association.

Here, details of the processing to control coloring to the setting itemand the text block according to the present modification example areexplained with reference to the flowchart in FIG. 9. The coloringcontrol processing shown in the flow in FIG. 9 is performed in place ofS 607 in the flow in FIG. 6 described previously.

At S901, the information on the character color of the character stringincluded in the selected text block for which the operation event hasbeen detected at S606 and the background color of the text block isacquired from the scanned document processing results storage unit 325via the scanned document processing application 311. Here, explanationis continued on the assumption that information on values (R, G, B=0, 5,5) indicating black as the character color and values (R, G, B=188, 213,254) indicating light blue as the background color is acquired.

At S902, information on the color used for coloring based on the itemcolor information at S605 (that is, the color allocated as the itemcolor of the selected metadata item and used for coloring to the itemcolor area) is acquired. It is assumed that allocation of “blue” to thefirst setting item, “green” to the second setting item, “yellow” to thethird setting item, and “red” to the fourth setting item is defined asthe default contents in the item color information in the presentmodification example. Here, in a case where the first “Customer Name” isselected among the three metadata items within the Metadata pane 411,the item color area is colored in “blue”, and therefore, in this case,information (for example, RGB values) indicating “blue” is acquired.

At next S903, the color allocated as the item color of the selectedmetadata item and the character color and the background color withinthe selected text block are compared and whether they are similar toeach other is determined. For example, as shown in FIG. 10A, the RGBvalues of the color allocated as the item color of the selected metadataitem and the RGB values of the character color and the background coloracquired at S901 are compared, respectively, and whether the differencetherebetween is within a predetermined threshold value is determined. Atthis time, it is desirable to set a threshold value by which, forexample, the light blue color or the purple color, whose hue is close tothat of the blue color, is determined to be “similar” to the blue color.Here, it is assumed that the blue color allocated as the item color ofthe selected metadata item “Customer Name” is determined to be similarto the light blue color, which is the background color of the selectedtext block. In a case where the color allocated as the item color forthe selected metadata item is determined to be similar to the charactercolor or the background color within the selected text block in thismanner, the processing advances to S904. On the other hand, in a casewhere it is determined that they are not similar, the processingadvances to S905.

At S904, processing to change the color currently allocated as the itemcolor of the selected metadata item to another different color definedin the item color information is performed. Specifically, theassociation between the item order and each color in the candidate colorinformation is changed and a different color is allocated to theselected metadata item. In a case of the above-described example, theorder of the selected metadata item “Customer Name” is the first.Consequently, as shown in FIG. 10B, the color is changed to the colorassociated with the fourth selected metadata item (in this case, “red”)whose order is the last and which is not used in the above-describedexample and at the same time, processing to associate “blue” with thefourth selected metadata item is performed. By exchanging the colorsbetween the first and fourth selected metadata items in this manner, itis possible to maintain the color association between the second andthird selected metadata items, and therefore, it is possible to suppressthe influence by the change to a minimum. After changing the color thatis the item color of the selected metadata item in this manner andupdating the item color information in accordance with the change, theprocessing returns to S903 and the similarity determination is performedagain. This processing is repeated until the determination resultsindicate that the color allocated as the item color is no longer similarto the character color or the background color within the selected textblock.

Then, at S905, the display of the color of the item color area of theselected metadata item is updated as needed. That is, in a case wherethe color allocated as the item color of the selected metadata item ischanged to another color at S904, coloring is performed in the newlychanged color.

At S906, the selected text block is colored by using the same color asthe color being displayed in the item color area of the selectedmetadata item. Due to this, the state of the UI screen as shown in FIG.8 described previously is brought about. In a case where coloring iscompleted, the processing returns to the flow in FIG. 6 and the displayprocessing at S608 is performed.

The above is the contents of the coloring control processing accordingto the modification example 2. As described in the first embodiment, itmay also be possible to perform control so that the coloring to the itemcolor area within the selected metadata item is not performed until thecoloring to the selected text block is performed and coloring to boththe item color area and the selected text block is performed at the sametiming.

MODIFICATION EXAMPLE 3

With an increase in the number of setting items, a case may occur whereit is difficult to select a color that is not similar to the charactercolor or the background color from among the colors defined in the itemcolor information and allocate the color. Consequently, an aspect isexplained as a modification example 3 in which a color that is the itemcolor of each setting item is determined and allocated without basingthe determination and allocation on the item color information.

Details of processing to control coloring to a setting item and a textblock according to the present modification example are explained withreference to the flowchart in FIG. 11. It is assumed that the coloringcontrol processing shown in the flow in FIG. 11 is performed in place ofS602 in the flow in FIG. 6 described previously. Then, at S605 in theflow in FIG. 6 in a case where the present modification example isapplied, coloring to the item color area is performed by using a colordetermined by processing, to be described later, in place of a colordetermined based on the item color information.

At S1101, the metadata item list is loaded. That is, in a case of thepresent modification example, item color information is not loaded. AtS1102 that follows, from the OCR processing results loaded at S601, theinformation on the character colors of the character strings included inall the text blocks within the page image and the background colors ofthe text blocks is acquired.

At S1103, in the character colors and the background colors acquired atS1102, a chromatic color whose frequency of appearance is high isspecified. For example, it is assumed that 100 text blocks exist withinthe page image and the black color is used as all the character colors.On the hand, as to the background color, it is assumed that there are 80text blocks for which the white color is used, 18 text blocks for whichthe light blue color is used, and two text blocks for which theyellowish green color is used. In this case, the chromatic color is thelight blue color and the yellowish green color, and therefore, the lightblue color whose number of times of use is large is specified as thechromatic color whose frequency of appearance is high. The reason blackand white, which are achromatic colors, are excluded is that they arenot suitable to the use as the item color aimed at identificationbecause black and white are used generally in a business form and thelike.

At S1104, based on the chromatic color that appears frequently, which isspecified at S1103, the item color for each metadata item is determined.Examples of a specific determination method are as follows.

A first method is a method that makes use of a hue circle. In this case,with a chromatic color that appears frequently being taken as areference, a color located at a distant position in the hue circle isdetermined as the item color for each metadata item. For example, in acase where the chromatic color that appears frequently is the light bluecolor and this is taken as a reference, the color located as the mostdistant position in the hue circle is the orange color, and therefore, acolor that is allocated as the item color is determined in apredetermined range with the orange color being taken as a center (forexample, range from yellow color to purple color). At this time, in acase where a plurality of setting items to which an item color is to beallocated exists and it is necessary to determine a plurality of colors,it is sufficient to select the plurality of colors so that, for example,the intervals in the hue circle are substantially equal in order todistinguish between the plurality of colors.

A second method is a method of determining a color that is not similarto the chromatic color that appears frequently as the item color among aplurality of candidate colors determined in advance. For example, ascandidate colors, five colors of red, green, yellow, blue, and purpleare determined in advance. Then, in a case where the chromatic colorthat appears frequently is the light blue color, the color is allocatedas the item color sequentially from the remaining four colors (red,green, yellow, purple) except for the blue color similar to the lightblue color. It is sufficient to determine the number of candidate colorsat this time in accordance with the format or the like of the businessform supposed as the processing target. Further, at the time ofdetermining the color to be excluded, it is sufficient to perform thesame similarity determination as that explained at S903 of themodification example 2.

The two methods explained above are examples and any method may be usedas long as the method is capable of allocating a color a user can easilydistinguish from another color as the item color with the chromaticcolor that appears frequently being taken as a reference.

FIG. 12 is a diagram explaining effects in a case where the presentmodification example is applied. In this example, the background colorwithin the ruled lines surrounding the character string of “XYZCorporation” is the light blue color, and therefore, three colors notsimilar to the light blue color are determined as the item colors andallocated. That is, to the three metadata items “Customer Name”,“Address”, and “Total Price”, the red color, the orange color, and theyellow color are allocated respectively as the item colors. Due to this,the colors used for coloring to three text blocks 1201 to 1203 areconspicuous, and therefore, it is made possible for a user to easilygrasp the association between the color and the selected text blockcorresponding to each selected metadata item.

The above is the contents of the coloring control processing accordingto the modification example 3. There may be a plurality of colorsspecified as chromatic colors that appear frequently. Further, it mayalso be possible to specify a chromatic color that appears frequently bytaking only the text blocks existing in a partial area, for example,such as the top portion of the page image, as a target in place of allthe text blocks existing within the page image. As in the modificationexample 2, it may also be possible to perform control so that coloringto the item color area within the selected metadata area is notperformed until coloring to the selected text block is performed andcoloring to both the item color area and the selected text block isperformed at the same timing.

As described above, according to the present embodiment, even in a casewhere a plurality of text blocks exists within the page image and aplurality of setting items (slave items) exists within the property paneon the UI screen having the preview pane and the property pane, it ispossible for a user to easily grasp the correspondence relationshiptherebetween.

Second Embodiment

In the first embodiment, the UI screen control on the assumption of acase where one-to-one association in which one text block is associatedwith one metadata item is performed is explained. Next, UI screencontrol capable of dealing also with a case where one-to-N associationin which the same text block is associated with a plurality of metadataitems is performed is explained as a second embodiment. Explanation ofthe contents in common to those of the first embodiment, such as thesystem configuration, is omitted and in the following, different pointsare explained mainly.

<UI Screen Control Processing>

The UI screen control processing according to the present embodiment,which is performed by the client application 351 of the client terminal121, is explained by using the flowchart in FIG. 13. Here, explanationis given on the assumption that association between each of the threemetadata items “Customer Name”, “Address”, and “Billing Name” and thetext block has been performed along the flow in FIG. 6 of the firstembodiment and the state of the UI screen shown in FIG. 14 is broughtabout.

On the UI screen in FIG. 14, at the top left of the text block 501within the page image 402, a label 1401 indicating the number ofmetadata items (here, “two”) with which the text block is associated isattached. This label is displayed at the timing at which the same textblock is associated with a plurality of metadata items (that is, at thepoint in time at which the number of metadata items becomes two ormore), but this is not limited and it may also be possible to displaythe label at the timing at which the text block is associated with oneof metadata items. Then, here, the text block 501 is the state of beingcolored in the default color (here, gray color) in a case whereassociation with a plurality of metadata items is performed. In contrastto this, a text block 1402 associated with only one metadata item“Address” is colored in the blue color indicated by the correspondingitem color area 511 b.

It is assumed that one of the display fields 510 a to 510 c of eachmetadata item within the Property pane 411 is selected by a user (touchoperation or the like with a finger) in a case where the UI screen is inthe state shown in FIG. 14. At S1301, the operation event, such as thetouch operation or the like by a user, is detected. Here, it is assumedthat the operation event for the display field 510 c of “Billing Name”is detected. At S1302 that follows, whether the text block associatedwith the metadata item selected by the user is also associated withanother metadata item is determined. In a case where the determinationresults are “No”, this processing is completed. On the other hand, in acase where the determination results are “Yes”, the processing advancesto S1303. Here, the text block 501 is associated with the metadata item“Billing Name” selected by the user and further, the text block 501 isalso associated with the metadata item “Customer Name”. Consequently,the determination results in this case is “Yes” and the processingadvances to S1303 as a result.

At S1303, the text block associated with the selected metadata item iscolored in the same color as the color of the item color area of theselected metadata item. Here, the color of the text block 501 changesfrom the gray color, which is the default color, to the yellow colorindicated by the item color area 511 c within the display field of themetadata item “Billing Name”. The UI screen shown in FIG. 15 shows thestate where the color of the text block 501 has changed from the graycolor to the yellow color accompanying the selection of the metadataitem “Billing Name”.

At next S1304, a button for cancelling the association between theselected metadata item and the text block is displayed. On the UI screenin FIG. 15, a “×” mark button 1501 that appears at the top right of thetext block 501 indicates the cancel button. At S1305 that follows,whether or not the touch operation or the like is performed to thecancel button is determined. In a case where the operation event, suchas the touch operation to the cancel button, is detected, the processingadvances to S1306 and in a case where no operation event is detected,this processing is exited.

At S1306, the association between the selected metadata item and thetext block is cancelled and the label indicating the number of metadataitems currently associated is updated. Specifically, as a result of thatthe association with the selected metadata item is cancelled, “one” issubtracted from the number of metadata items indicated by the label. Inthe example in FIG. 15, the numerical value of the label 1401 changesfrom “two” to “one”. Further, accompanying this, the display fieldcorresponding to the selected metadata item is initialized and the blankstate is brought about. Here, the selected metadata item is “BillingName”, and therefore, the corresponding display field 510 c isinitialized.

The above is the contents of the UI screen control processing in theclient application 351 according to the present embodiment.

As above, according to the present embodiment, even in a case where thesame text block is associated with a plurality of setting items (slaveitems), it is possible for a user to grasp the correspondencerelationship between a slave item and a text block both accurately andeasily.

Other Embodiments

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

According to the technique of the present disclosure, on a UI screenhaving a preview pane and a property pane, it is possible for a user toeasily grasp a correspondence relationship between a text block and asetting item.

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the disclosure is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Applications No.2020-007518 filed Jan. 21, 2020, and No. 2020-031689 filed Feb. 27,2020, which are hereby incorporated by reference wherein in theirentirety.

1.-17. (canceled)
 18. An image processing system, comprising: a memorythat stores a program; and a processor that executes the program toperform: displaying a user interface screen for setting a property to animage, wherein the user interface includes a preview pane that is anarea preview-displaying the image and a property pane that is an area towhich a setting value of each setting item of the property is input; andcontrolling the user interface screen to further display, on the previewpane, a label for a first text block included in the image, wherein thelabel indicates a number of setting items associated with the first textblock.
 19. The image processing system according to claim 18, whereinthe label is displayed for the first text block on the preview paneafter the first text block is associated with two or more setting itemsdisplayed in the property pane.
 20. The image processing systemaccording to claim 18, wherein the label is displayed for the first textblock on the preview pane after the first text block is associated withat least one of setting items displayed in the property pane.
 21. Theimage processing system according to claim 18, wherein the processorexecutes the program to further perform: coloring the first text blockwith a default color if the first text block is associated with two ormore setting items and any of the associated two or more setting itemsis not being in a selected state; and coloring, when a user selects oneof the associated two or more setting items in the property pane, thefirst text block with a color corresponding to the selected one settingitem.
 22. The image processing system according to claim 18, wherein theprocessor executes the program to further perform: coloring, if a secondtext block in the image is associated with only one second setting item,both of at least a part of a field of the second setting item displayedin the property pane and the second text block in the imagepreview-displayed in the preview pane with a same or similar color. 23.The image processing system according to claim 18, wherein the processorexecutes the program to further perform: coloring, in a case where auser selects one of text blocks in the image preview-displayed in thepreview pane in a selected state where one setting item which is notassociated with any text block is selected in the property pane, theselected one text block with a color that is same as or similar to thecolor allocated to the one setting item in the selected state.
 24. Acontrol method comprising: displaying a user interface screen forsetting a property to an image, wherein the user interface includes apreview pane that is an area preview-displaying the image and a propertypane that is an area to which a setting value of each setting item ofthe property is input; and further displaying, on the preview pane, alabel for a first text block included in the image, wherein the labelindicates a number of setting items associated with the first textblock.
 25. The control method according to claim 24, wherein the labelis displayed for the first text block on the preview pane after thefirst text block is associated with two or more setting items displayedin the property pane.
 26. The control method according to claim 24,wherein the label is displayed for the first text block on the previewpane after the first text block is associated with at least one ofsetting items displayed in the property pane.
 27. The control methodaccording to claim 24, further comprising: coloring the first text blockwith a default color if the first text block is associated with two ormore setting items and any of the associated two or more setting itemsis not being in a selected state; and coloring, when a user selects oneof the associated two or more setting items in the property pane, thefirst text block with a color corresponding to the selected one settingitem.
 28. The control method according to claim 24, further comprising:coloring, if a second text block in the image is associated with onlyone second setting item, both of at least a part of a field of thesecond setting item displayed in the property pane and the second textblock in the image preview-displayed in the preview pane with a same orsimilar color.
 29. The control method according to claim 24, furthercomprising: coloring, in a case where a user selects one of text blocksin the image preview-displayed in the preview pane in a selected statewhere one setting item which is not associated with any text block isselected in the property pane, the selected one text block with a colorthat is same as or similar to the color allocated to the one settingitem in the selected state.
 30. A non-transitory computer readablestorage medium storing a program for causing a computer to perform:displaying a user interface screen for setting a property to an image,wherein the user interface includes a preview pane that is an areapreview-displaying the image and a property pane that is an area towhich a setting value of each setting item of the property is input; andfurther displaying, on the preview pane, a label for a first text blockincluded in the image, wherein the label indicates a number of settingitems associated with the first text block.
 31. The non-transitorycomputer readable storage medium according to claim 30, wherein thelabel is displayed for the first text block on the preview pane afterthe first text block is associated with two or more setting itemsdisplayed in the property pane.
 32. The non-transitory computer readablestorage medium according to claim 30, wherein the label is displayed forthe first text block on the preview pane after the first text block isassociated with at least one of setting items displayed in the propertypane.
 33. The non-transitory computer readable storage medium accordingto claim 30, wherein the program causes the computer to further perform:coloring the first text block with a default color if the first textblock is associated with two or more setting items and any of theassociated two or more setting items is not being in a selected state;and coloring, when a user selects one of the associated two or moresetting items in the property pane, the first text block with a colorcorresponding to the selected one setting item.
 34. The non-transitorycomputer readable storage medium according to claim 30, wherein theprogram causes the computer to further perform: coloring, if a secondtext block in the image is associated with only one second setting item,both of at least a part of a field of the second setting item displayedin the property pane and the second text block in the imagepreview-displayed in the preview pane with a same or similar color. 35.The non-transitory computer readable storage medium according to claim30, wherein the program causes the computer to further perform:coloring, in a case where a user selects one of text blocks in the imagepreview-displayed in the preview pane in a selected state where onesetting item which is not associated with any text block is selected inthe property pane, the selected one text block with a color that is sameas or similar to the color allocated to the one setting item in theselected state.